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Exact Reconstruction Conditions for 
Regularized Modified Basis Pursuit 



Wei Lu and Namrata Vaswani 



Abstract 



In this work, we obtain sufficient conditions for exact recovery of regularized modified basis pursuit 
(reg-mod-BP) and discuss when the obtained conditions are weaker than those for modified compressive 
I sensing or for basis pursuit (BP). The discussion is also supported by simulation comparisons. Reg-mod- 

BP provides a solution to the sparse recovery problem when both an erroneous estimate of the signal's 



^ ■ support, denoted by T, and an erroneous estimate of the signal values on T are available. 



Index Terms 

Compressive sensing, modified-CS, partially known support, sparse reconstruction 

I. Introduction 

c/3 ■ In this work, we obtain sufficient conditions for exact recovery of regularized modified basis pursuit 

(reg-mod-BP) and discuss when the obtained conditions are weaker than those for modified compressive 
^ . sensing lO or for basis pursuit (BP) lH, H. Reg-mod-BP was briefly introduced in our earlier work Q 

^ . as a solution to the sparse recovery problem when both an erroneous estimate of the signal's support, 

l/^ . denoted by T, and an erroneous estimate of the signal values on T, denoted by (/i)T> are available. 

The problem is precisely defined in Sec. I-A. Reg-mod-BP, given in ([TT]) . tries to find a vector that is 
sparsest outside the set T among all solutions that are close enough to (/t)T on T and satisfy the data 
O . constraint. In practical applications, T and (/i)^ may be available from prior knowledge, or in recursive 

reconstruction applications, e.g. recursive dynamic MRI iQ, IJl, recursive compressive sensing (CS) 
based video compression |[6l, Q, or recursive projected CS (ReProCS) HI, Q based video layering, one 
^ . can use the support and signal estimate from the previous time instant for this purpose. 

H I Basis pursuit (BP) was introduced in [31 as a practical (polynomial complexity) solution to the problem 

of reconstructing a sparse m x 1 vector, x, with support denoted by N, from an n x 1 measurements' 
vector, y := Ax, when n < m. BP solves the following convex (actually linear) program: 

rmn ||/3||i subject to y = Af3 (1) 

The recent CS literature has provided strong exact recovery results for BP that are either based on the 
restricted isometry property (RIP) lH, ifTOl or that use the geometry of convex poly topes to obtain "exact 
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recovery thresholds" on the n needed for exact recovery with high probabiUty lITll . |[T2l . BP is often just 
referred to as CS in recent works and our work also occasionally does this. 

In recent work ||2l, we introduced the problem of sparse reconstruction with partial and partly erroneous 
support knowledge, denoted by T, and proposed a solution called modified compressive sensing (mod- 
CS). We obtained exact reconstruction conditions for mod-CS and showed when they are weaker than 
those for BP. Mod-CS tries to find the solution that is sparsest outside the set T among all solutions of 
y = Aj3, i.e. it solves 



Ideally the above should be referred to as mod-BP, but since we used the term mod-CS when we introduced 
it, we will retain it here. Similar problems were also studied in parallel work by von Borries et al. | [T3T| 
and Khajehnejad et al. ||T4l . In |[T4l . the authors assumed a probabilistic prior on the support, solved the 
following weighted £i problem and obtained exact recovery thresholds similar to those in lfT2l : 



In another related work |fT5l . Wang et al. showed how to iteratively improve recovery of a single signal 
by solving BP in the first iteration, obtaining a support estimate, solving Q with this support estimate 
and repeating this. They also obtained exact recovery guarantees for a single iteration. 

Another related idea is CS-diff or CS-residual which recovers the residual signal x — fi hy solving ([T|l 
with y replaced by y — Afi. This is related to our earlier least squares CS-residual (LS-CS) and Kalman 
filtered CS (KF-CS) ideas ||5l, |[T6l . However, as explained in lH, the residual signals using all these 
methods have a support size that is equal to or slightly larger than that of x (except if {jl)T = ^t)- As 
a result, these do not achieve exact recovery with fewer measurements. The limitations of some other 
variants of this are also discussed in detail in lUTl . Reg-mod-BP may also be interpreted as a Bayesian 
or a model-based CS approach. Recent work in this area includes iflSl . |[T9l . EOl . 

This paper is organized as follows. We introduce reg-mod-BP in Sec. II. In Sec III, we obtain the exact 
reconstruction result, discuss its implications and give the key lemmas leading to its proof. Simulation 
comparisons are given in Sec. IV and conclusions in Sec. V. 

A. Notation and Problem Definition 

For a set T, T'^ = {i G [1, ...,m],i ^ T}. is the empty set. We use |.| to denote the cardinality of a 
set. The same notation is also used for the absolute value of a scalar. The meaning is clear from context. 

For a vector b, {b)T, or just hr, denotes a sub-vector containing the elements of b with indices in 
T. \\b\\h means the £k norm of the vector b. The notation 6 ^ (6 ^ 0) means that each element of 
the vector b is greater than or equal to (strictly greater than) zero. Similarly 6 ^ (6 -< 0) means each 
element is less than or equal to (strictly less than) zero. We define the sign pattern, sgn(6) as: 



We use ' for matrix transpose. For a matrix A, At denotes the sub-matrix containing the columns of A 
with indices in T. Also, \\A\\ := maxa-^o TraT~ induced 2 norm. 



min ||/3t<^||i subject to y = Afi 



(2) 



min Wfix'-- ||i + 7||/5t||i subject to y = Afi 



(3) 




(4) 
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Our goal is to solve the sparse reconstruction problem, i.e. reconstruct an m-length sparse vector, x, 
with support, A^, from an n < m length measurement vector, 

y := Ax (5) 

when an erroneous estimate of the signal's support, denoted by T; and an erroneous estimate of the signal 
values on T, denoted by {fi)T, are available. The support estimate, T, can be rewritten as 

T = iV U Ae \ A, where A := \ T and Ae := T \ iV (6) 

are the errors (A contains the misses while Ag contains the extras) in the support estimate. 
The signal value estimate is assumed to be zero along T'^, i.e.. 



(A)t 



and it satisfies 

{fl)T = {x)t + i^, with ||i/||oo <p (7) 

The restricted isometry constant (RIC) Ml/. Sg, for A, is defined as the smallest positive real number 
satisfying (1 — (5s)||c||2 < IIA5CII2 < (1 + '^s)||c||2 for all subsets S of cardinality l^l < s and all 
real vectors c of length The restricted orthogonality constant (ROC) /@/, ^51,53, is defined as the 
smallest positive real number satisfying \ci' At^' At2C2\ < ^si,s2l|ci||2||c2||2 for all disjoint sets Ti,T2 
with |Ti| < si, IT2I < S2 and si + S2 < m, and for all vectors ci, C2 of length |Ti|, \T2\ respectively. 
Both 6s and Og-^^^s^ '^re non-decreasing functions of s and of si, S2 respectively IH. 

We will frequently use the following functions of the RIC and ROC of A in Sec. Ill: 

ak{s,s) := (8) 

1 - - 

Kk{u) := (9) 

For the matrix A, and for any set S for which As' As is full rank, we define the matrix M{S) as 

MiS):=I-AsiAs'Asr^As' (10) 



II. Regularized Modified Basis Pursuit 
Mod-CS given in Q puts no cost on f^T and no explicit constraint except y = Afi. Thus, when very 
few measurements are available, Pt can become larger than required in order to satisfy y = A(3 with the 
smallest H/Jtc A similar, though less, bias will also occur with Q when 7 < 1. However, if a signal 
value estimate on T, (/i)r> is also available, one can use that to constrain f^r- One way to do this, as 
suggested in is to add A||/3t — AtHI to the mod-CS cost. However, as we saw from simulations, while 
this does achieve lower reconstruction error, it cannot achieve exact recovery with fewer measurements 
(smaller n) than mod-CS 13. The reason is it puts a cost on the entire £2 distance from {il)T and so 
encourages elements on the extras set, Ag, to be closer to (/i)A, which is nonzero. 
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On the other hand, if we instead use the £00 distance from (/i)T> and add it as a constraint, then, at 
least in certain situations, we can achieve exact recovery with a smaller n than mod-CS. Thus, we study 

rmn ||/3t<=||i, subject to y = AP and \\(3t — /^tIIoo < P (11) 

and call it reg-mod-BP. We see from simulations, that whenever one or more of the inequality constraints 
are active at X, i.e. \xi—jli\ = pforsomei G T, diiP does achieve exact recovery with fewer measurements 
than mod-CS. We use this observation to derive a better exact recovery result belowj^. 

III. Exact Reconstruction Conditions 
In this section, we obtain exact reconstruction conditions for reg-mod-BP by exploiting the above fact. 
We give the result and discuss its implications below in Sec III-A. The key lemmas leading to its proof 
are given in Sec. III-B and the proof outline in Sec. III-C. 

A. Exact Reconstruction Result 

Let us begin by defining the two types of active sets (set of indices for which the inequality constraint 
is active), and Ta_, and the inactive set. Tin, as follows. 





:= {. 


G T : 


Xi /i^ 


= P}, 




:= {. 


G T : 


Xi /i^ 


= -p} 




:= {i 


G T : 




\<P} 



In the result below, we try to find the sets Ta+g C Ta+ and Ta-g C Ta_ so that |Ta+g| + |Ta_g| is maximized 
while Ta+g and Ta.g satisfy certain constraints. We call these the "good" sets. We define the "bad" subset 
of T, as Tfc := T \ (Ta+g U Ta.g). As we will see, the smaller the size of this bad set, the weaker are our 
exact recovery conditions. 

Theorem 1 (Exact Recovery Conditions): Consider recovering a sparse vector, x, with support N, from 
y := Ax by solving (fTTI) . The support estimate, T, and the misses and extras in it. A, Ag, satisfy Q. 
The signal estimate, fi, satisfies ([7]), i.e. \\xt — firWoo P- Define the sizes of the sets T and A as 

k := |T|, u := |A|. (13) 

The true x is the unique minimizer of ([TT]) if 

1) 6k+u < 1, S2u + + < 1, and 

'One can also try to constrain the £2 distance instead of the £oa distance. When the £2 constraint is active, one should again 
need a smaller n for exact recovery. When we check this via simulations, this does happen, but since it is at most one active 
constraint, the reduction in n required is small compared to what is achieved by dl It and hence we do not study this further. 
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2) ak{2u,u) + ak^{u,u) < 1 where 

Tb := r\(r,+gUTa.g), and 

h ■= \Tb\ 

{Ta+g,Ta_g} = arg max {\f.^+^\ + |ra_g|) subject to 
T C T T C T 

^ a+g ^ a+ ) a-g ^ a- > 

^i'-u; > V i G fa+g, and ^/it; < V i G fa.g, (14) 
where 

w := M(7],)^A(AA'M(rb)AA)"^sgn(xA), 

Tb:=T\(fa+gUTa.g), 

M(5) is specified in (fTOl) . s) is defined in ([8]l, and the sets Ta+, Ta_ are defined in (fT2l ). ■ 
Notice that afc(s, s) is a non-decreasing function of k. Since kh = k — |Ta+g| — |ra_g|, thus, finding the 
largest possible sets Ta+g and Ta_g ensures that the condition ak{2u,u) + ak^{u,u) < 1 is the weakest. 
The reason for defining Ta+g and Ta_g in the above fashion will become clear in the proof of Lemma |2] 
Notice also that the first condition of the above result ensures that 6k < 1- Since |Tf,| < k, thus, 
AfjAj.^ is positive definite and thus invertible. Thus M{T}j) is always well defined. The first condition 
also ensures that ak{2u,u) > 0. Since ki^ < k, and since 6s and 9si,s2 non-decreasing functions of 
s,si,S2, it also ensures that ak^{u,u) > 0. 

Remark 1 (Applicability): A practical case where some of the inequality constraints will be active with 
nonzero probability is when dealing with quantized signals and quantized signal estimates. If the range 
of values that the signal estimate can take given the signal (or vice versa) is known, the smallest choice 
of p is easily computed. We show some examples in Sec. IV. In general, even if just the range of values 
both can take is known, we can compute p. The fewer the number values that Xi — fii can take, the 
larger will be the expected size of the active set, Ta := T^^ U T^.. Also, the condition ([T4ll will hold for 
non-empty Tg := Ta+g U Ta.g with nonzero probability. 

Some real applications where quantized signals and signal estimates occur are recursive CS based video 
compression [61, Q (the original video itself is quantized) or in recursive projected CS (ReProCS) IHl, ||9l 
based moving or deforming foreground objects' extraction (e.g. a person moving towards a camera) from 
very large but correlated noise (e.g. very similar looking but slowly changing backgrounds), particularly 
when the videos are coarsely quantized (low bit rate). A common example where low bit rate videos 
occur is mobile telephony applications. In any of these applications, if we know a bound on the maximum 
change of the sparse signal's value from one time instant to the next, that can serve as p. 

Remark 2 (Comparison with BP, mod-CS, other results): The worst case for Theorem 1 is when both 
the sets Ta+g and Ta.g are empty either because no constraint is active (Ta+ and Ta_ are both empty) or 
because (fT4l ) does not hold for any pair of subsets of Ta+ and Ta_. In this case, we have kb = k and 
so the required sufficient conditions are the same as those of mod-CS ||2j Theorem 1]. A small extra 
requirement is that x satisfies ([7]). Thus, in the worst case. Theorem 1 holds under the same conditions 
on A (needs the same number of measurements) as mod-CS 111. In 121, we have already argued that 
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the mod-CS result holds under weaker conditions than the results for BP ||4l, lITOl as long as the size of 
the support errors, |A|, |Ae|, are small compared to the support size, |A^|, and hence the same can be 
said about Theorem 1. For example, we argued that when |A| = |Ae| = 0.02|A^| (numbers taken from a 
recursive dynamic MRI application), the mod-CS conditions are weaker than those of BP. Small |A|, |Ae| 
is a valid assumption in recursive recovery applications like recursive dynamic MRI, recursive CS based 
video compression, or ReProCS based foreground extraction from large but correlated background noise. 
Moreover, if some inequality constraints are active and (fT4l ) holds, as in case of quantized signals and 
signal estimates. Theorem 1 holds under weaker conditions on A than the mod-CS result. 
As noted by an anonymous reviewer, our exact recovery conditions require knowledge of x. However 
this is an issue with many results in sparse recovery, e.g. llTI . and especially those that use more prior 
knowledge, e.g. |[T8l . 

Remark 3 (Small reconstruction error): The reconstruction error of reg-mod-BP is significantly smaller 
than that of mod-CS, weighted £i or BP, even when none of the constraints is active, as long as p is small 
(see Table HID ). On the other hand, the exact recovery conditions do not depend on the value of p, but 
only on the size of the good subsets of the active sets. This is also observed in our simulations. In Table 
Hill we show results for p = 0.1. Even when we tried p = 0.5, the exact reconstruction probability or the 
smallest 7i needed for exact reconstruction remained the same, but the reconstruction error increased. 

Remark 4 ( Computation complexity ): Finding the best Ta+g and Ta-g requires that one check all possible 
subsets of Ta+ and T^. and find the pair with the largest sum of sizes that satisfies ([141 ). To do this, one 
would start with Ta+g = Ta+, Ta-g = Ta.; compute TJ, and w and check if (1141 ) holds; if it does not, remove 
one element from Ta+g and then check (1141 ): then remove an element from Ta.g and check ([T4l ): keep doing 
this until one finds a pair for which ([T4l ) holds. In the worst case, one will need to check ([141 ) 2l^''+l+l^''-l 
times. However, the complexity of computing the RIC 6\x\ or any of the ROC's is anyway exponential 
in |T| and \T\ > jTa+j + |Ta_|. In summary, computing the conditions of Theorem 1 has complexity that 
is exponential in the support size, but the same is true for all sparse recovery results that use the RIC. 
We should mention though that, for certain random matrices, e.g. random Gaussian, there are results that 
upper bound the RIC values with high probability, e.g. see IH. However, the resulting bounds are usually 
quite loose. 

B. Proof of Theorem 1: Key Lemmas 

Our overall proof strategy is similar to that of jUl for BP and of ||2l for mod-CS. We first find a set of 
sufficient conditions on an n x 1 vector, w, that help ensure that x is the unique minimizer of ([TTl ). This 
is done in Lemma [T] Next, we find sufficient conditions that the measurement matrix A should satisfy 
so that one such w can be found. This is done in an iterative fashion in the theorem's proof. The proof 
uses Lemma |2] at the zeroth iteration, followed by applications of Lemma [3] at later iterations. 

To obtain the sufficient conditions on w, as suggested in lH, we first write out the Karush-Kuhn-Tucker 
(KKT) conditions for x to be a minimizer of (ITTl ) |[22l Chapter 5]. By strengthening these a little, we 
get a set of sufficient conditions for x to be the unique minimizer. The necessary conditions for x to be 
a minimizer are: there exists an n x 1, vector w (Lagrange multiplier for the constraints in y = Ax), a 
iTa+l X 1 vector, Ai, and a |Ta_| x 1 vector, A2, such that (s.t.) 

1) every element of Ai and A2 is non-negative, i.e. Ai ^ and A2 h 0, 
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2) AtJw = 0, AtJw = Ai, AtJw = -A2, Aa'w = sgn(xA), and P(tuA)-'''^I|oo < 1- 
As we will see in the proof of Lemma [T] strengthening ||^(tuA)='^I|oo < 1 to ||^(ruA)<='w;||oo < 1, 
keeping the other conditions the same, and requiring that d^+u < 1 gives us a set of sufficient conditions. 

Lemma 1: Let x be as defined in Theorem 1. x is the unique minimizer of (fTTl) if 6k+u < 1 and if 
we can find an n x 1 vector, w, s.t. 

1) AtJw = 0, AtJw h 0, AtJw ^ 0, 

2) Aa'w = sgn(xA), 

3) \A/w\ < 1 for all j ^ T U A. 

Recall that T^+, T.^_ and are defined in ([T2l ) and k,u in Theorem 1. ■ 
Proof: The proof is given in Appendix [A] 

Notice that the first condition is weaker than that of Lemma 1 of mod-CS lO (which requires Ajw = 
0), while the other two are the same. Next, we try to obtain sufficient conditions on the measurement 
matrix, A (on its RIC's and ROC's) to ensure that such a w can be found. This is done by using Lemmas 
[2] and [3] given below. Lemma |2] helps ensure that the first two conditions of Lemma [T] hold and provides 
the starting point for ensuring that the third condition also holds. Then, Lemma [3] applied iteratively helps 
ensure that the third condition also holds. 

Lemma 2: Assume that k + u < m. Let s be such that k + u + s < m. \i 6u + 5kf, + O"^^ ^ < 1, then 
there exists an n x 1 vector w and an "exceptional" set, E, disjoint with T U A, s.t. 

1) At.'w = 0, AtJw ^ 0, AtJw ^ 0, 

2) Aa'w = sgn(xA), 

3) 1^1 <s, \\AE'w\\2<akJu,s),Jl, \Ajw\ < """^'"'^ ^ Vj^TuAuS, 

4) ll^lla < Kkju)^. 

Recall that 0^(5, s), Kk{s) are defined in ([8]), dill and T^+g, Ta_g, T^, k^,, k and u in Theorem[T] ■ 

Notice that because we have assumed that 5u + f^fc^ + ^fc^ « < o-kju, s) and Kju) are positive. We 
call the set E an "exceptional" set, because except on the set E C (TU A)'^, everywhere else on (TU A)"^, 
lAj'wl is bounded. This notion is taken from ||4l. Notice that the first two conditions of the above lemma 
are one way to satisfy the first two conditions of Lemma [T] since = Tin U {T^+ \ Tk+g) U (Ta_ \ TJ. 

Proof: The proof is given in Appendix E We let w = M{TiJ)Aa{Aa' M{Tb)AA)~^^gr\{xA)- Since 
the good sets Ta+g, Ta_g are appropriately defined (see (fT4l)). the first two conditions hold. The rest of the 
proof bounds \\w\\2, and finds the set C (T U A)^ of size \E\ < s so that l^/tL"! is bounded for all 
i ^ T U A U £^ and also ||^£;'tx)||2 is bounded. 

Lemma 3 (Lemma 2 of f^): Assume that k < m. Let s, s be such that k + s + s < m. Assume that 
+ + (^k s < ^- be a set that is disjoint with T, of size \Tfi\ < s and let c be a IT^^I x 1 

vector. Then there exists an n x 1 vector, w, and a set, E, disjoint with T U Td, s.t. (i) At'w = 0, (ii) 
AtJw = c, (iii) \E\ < s, \\Ae'w\\2 < ak{s, s)\\c\\2, \Ajw\ < ^^^||c||2, Vj ^ TuTaUE, and (iv) 
ll^lb < Kk{s)\\c\\2. 

Recall that ak{s,s), Kk{s) are defined in ([8]l, dill, and k,u in Theorem 1. ■ 
Proof: The proof of Lemma |3l is given in lH and also in Appendix C of |[23l . 
Notice that because we have assumed that 5s + 6k + Ol ^ < I, ak{s, s) and Kk{s) are positive. 
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2K 


BP 










4 





0.18 


0.16 


0.64 


N-RMSE(0.15m) 


4 


1.011 


0.059 


0.060 


0.029 


ncxact(l) 


4 


0.39m 


0.21m 


0.21m 


0.18m 


Pexact (0.15m) 


10 





0.18 


0.16 


0.39 


N-RMSE(0.15m) 


10 


1.011 


0.059 


0.060 


0.032 


ncxact(l) 


10 


0.4m 


0.21m 


0.21m 


0.20m 



TABLE I: Quantized signals and signal estimates. Recall that k = \T\ ^ 26. For 2K = 4, the expected sizes of Ta, 
Tg and Tf, are E[\Ta\] = 10.01, E[\Tg\] = 5.27 and E[\Tb\] = 20.73. For 2K = 10, mTa\] = 4.28, E[|Tg|] = 2.3 
and E[|Tfc|] = 23.7. 

C. Proof Outline of Theorem [7] 

The proof is very similar to that of ||2l. Hence we give only the outline here. The complete proof is in 
ll23l . At iteration zero, we apply Lemma [2] with s = u, to get a wi and an exceptional set T^ i, disjoint 
with TU A, of size less than u. Lemma |2] can be applied because ki, < k and condition [T] of the theorem 
holds. At iteration r > 0, we apply Lemma[3]with Td = AuT^r (so that s = 2u), ca = 0, ct^ = Axjwr 
and s = n to get a Wr+i and an exceptional set Td,r+i disjoint with T U A U of size less than u. 
Lemma |3]can be applied because condition [T] of the theorem holds. Define w := We 
then argue that if condition |2] of the theorem holds, w is well-defined and satisfies the conditions of 
Lemma [T] Applying Lemma [T] the result follows. 

IV. Numerical Experiments 

In this section, we show two types of numerical experiments. The first simulates quantized signals 
and signal estimates. This is the case where some constraints are active with nonzero probability. The 
good set, Tg = Ta+g U Ta-g is also non empty with nonzero probability. Hence, for a given small enough 
n, reg-mod-BP has significantly higher exact reconstruction probability, Pexactin), as compared to both 
mod-CS im and weighted £i |[T4l and much higher than that of BP lU, lH. Alternatively, it also requires 
a significantly reduced n for exact reconstruction with probability one, nexact(l)- In computing Pexact('^) 
we average over the distribution of x, T and fi, as also in 121, ||4l. All numbers are computed based on 
100 Monte Carlo simulations. To compute nexact(l)> we tried various values of n for each algorithm and 
computed the smallest n required for exact recovery always (in all 100 simulations). 

We also do a second simulation where signal estimates are not quantized. 

In the following steps, the notation z ~ discrete-uniform(ai, 02, . . . a„) means that z is equally likely 
to be equal to ai, 02, ... or a„. We use ita as short for +a, —a. Also, z ~ uniform(a, b) generates a 
scalar uniform random variable in the range [a, b]. The notation Xi '~ P for alH G 5 means that, for all 
i £ S, each Xi is identically distributed according to P and is independent of all the others. 

For the quantized case, x was an m = 256 length sparse vector with support size |A^| = 0.1m = 26 
and support estimate error sizes u = \A\ = |Ae| = CljA'^l = 3. We generated the matrix A once as an 
n X m random Gaussian matrix (generate an n x m matrix with i.i.d zero mean Gaussian entries and 
normalize each column to unit £2 norm). The following steps were repeated 100 times. 
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BP 


mod-CS 


weighted £i 


Reg-mod-BP 


Pcxact (0.15m) 





0.26 


0.26 


0.57 


N-RMSE(0.15m) 


0.967 


0.152 


0.152 


0.082 


ncxact(l) 


0.4m 


0.21m 


0.21m 


0.20m 



TABLE II: Quantized signals and signal estimates: case 2. Recall that k — \T\ — 26. The expected sizes of Ta, 
Tg and are '&[\Ta\] = 9.02, E[|Tg|] = 4.58 and E[|r6|] = 21.42. 





BP 


mod-CS 


weighted h 


Reg-mod-BP 


Pcxact(0.18m) 





0.87 


0.87 


0.87 


N-RMSE(0.18m) 


0.961 


0.0175 


0.0177 


0.0123 


N-RMSE(O.llm) 


1.05 


0.179 


0.175 


0.0635 


'^exact(l) 


0.39m 


0.21m 


0.21?n 


0.21?n 



TABLE III: The non quantized case. 



1) The support set, N, of size |A^|, was generated uniformly at random from The support 
misses set. A, of size u, was generated uniformly at random from the elements of N. The support 
extras set, Ae, also of size u, was generated uniformly at random from the elements of N^. The 
support estimate, T = U Ag \ A and thus \T\ = \N\= 26. 

2) We generated Xi *~ discrete-uniform ( it 1) for i G A^ n T; Xi discrete-uniform(ibO.l) for i S 
A, and Xj = for i G A^'^. XNr\T and xa are also independent of each other. We generated 
fiT = XT + y where Vi *~ discrete-uniform(0, it-^, ib2-|^, • • • it p) for i G T n N and Ui *~ 
discrete-uniform(±-^, ±2;^, • • • it p) for i G Ag. We used p = 0.1 and tried two choices of K. 
Notice that, for a given K, the number of equally likely values that Xi — /zj for i G T can take are 
roughly 2K + 1 {2K when i G Ag). The constraint is active when — /tj is equal to it p. Thus, 
the expected size of the active set is roughly 2k+i 1^1- 

3) We generated y = Ax. We solved reg-mod-BP given in (ITTI ) with p = 0.1; BP given in ([T|); mod-CS 
given in and weighted ii given in ^ with various choices of 7: [0.1 0.05 0.01 0.001]. We 
used the CVX optimization package, http://www.stanford. edu/boyd/cvx/[ which uses primal-dual 
interior point method for solving the minimization problem. 

We computed Pexact("-) as the number of times x was equal to x ("equal" was defined as ||x — x||2/||x||2 < 
10^5) divided by 100. For weighted £1, we computed Pexactin) for each choice of 7 and recorded the 
largest one. This corresponded to 7 = 0.1. We tabulate results in Table H In the first row, we record 
Pexact(0.15m) for all the methods, when using K = 2. We also record the Monte Carlo average of the sizes 
of the active set \Ta\ = \T^+UTi,_\; of the good set, \Tg\ = |ra+gUTa_g| and of the bad set jTbl = A; - IT^]. 
In the second row, we record the normalized root mean squared error (N-RMSE). In the third row, we 
record nexact(l)- In the next three rows, we repeat the same things with K = 5. 

As can be seen, \Tg\ is about half the size of the active set, \Ta\. As K is increased, \Ta\ and hence 
\Tg\ reduces {\Ti,\ increases) and thus pexact(0.15m) decreases and nexact(l) increases. Also, for mod-CS 
and weighted £1, Pexact (0* is significantly smaller than for reg-mod-BP, while ?^exact 

(1) is larger. 

Next, we simulated a more realistic scenario - the case of 3-bit quantized images (both x and p take 
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integer values between to 7). Here again m = 256, |A^| = 0.1m = 26, and n = |A| = | Ae| = 0.1|A^| = 
3. The sets N, A, Ag and T were generated as before. We generated Xi *~ discrete-uniform(3, 4, ... 7) 
for i G n T; ~ discrete-uniform(l, 2) for i G A; and Xj = for i G N'^. Also, fix = clip(x7- + u) 
where Ui discrete-uniform(— 2, —1, 0, 1, 2) for i G T n A^; and i/j ~ discrete-uniform(— 2, —1, 1, 2) for 
i G Ag. Also clip(z) clips any value more than 1 to 1 and any value less than zero to zero. Clearly, in 
this case p = 2. We record our results in Table JI] Similar conclusions as before can be drawn. 

Finally, we simulated the non-quantized case. We used m = 256, |A^| = 0.1m = 26, and n = |A| = 
|Ae| = O.ljA^I = 3. We generated Xj '~ discrete-uniform ( it 1) fori G NCiT; Xj *~ discrete-uniform(ibO.l) 
for i G A, and Xj = for i G N'^. The signal estimate, (It = xt + i' where f j *~ uniform(— p, p) with 
p = 0.1. We tabulate our results in Table Hill Since is a real vector (not quantized), the probability of 
any constraint being active is zero. Thus, as expected, Pexact and ngxact are the same for reg-mod-BP and 
mod-CS and weighted , though significantly better than BP. However, the N-RMSE for reg-mod-BP is 
significantly lower than that for mod-CS and weighted li also, particularly when n = 0.11m. 

V. Conclusions 

In this work, we obtained sufficient exact recovery conditions for reg-mod-BP, (fTTI ). and discussed 
their implications. Our main conclusion is that if some of the inequality constraints are active and if 
even a subset of the set of active constraints satisfies certain conditions (given in ([T4l)). then reg-mod-BP 
achieves exact recovery under weaker conditions than what mod-CS needs. A practical situation where 
this would happen is when both the signal and its estimate are quantized. In other cases, the conditions 
are only as weak as those for mod-CS. In either case they are much weaker than those for BP as long 
as T is a good support estimate. From simulations, we see that even without any active constraints, the 
reg-mod-BP reconstruction error is much lower than that of mod-CS or weighted li. 

Appendix 

A. Proof of Lemma [7] 

Denote a minimizer of ([TTI l by /3. Since y = Ax and x satisfies dTJl, x is feasible for ([TTI l. Thus, 

\\PtA\i < I^T-lli = I^aIIi (15) 
Next, we use the conditions on w given in Lemma [T] and the fact that x is supported on C T U A to 
show that ||/3tH|i > l|2^T<^||i and hence ||xt=||i = ||/3rH|i- Notice that 

Wt^ 111 = E l^i + Pi - ^^-1 + E l^jl ^ E 1^^- + l^i - ^il + E "^'^iPi (^6) 
jeA i^TuA jeA j^TuA 

> J]sgn(xj)(xj + (^j-xj))+ w'Aj{f3j-Xj) (17) 
jeA j^TuA 

= \\xa\\i + Yw' Aj {(3 j - Xj) = \\xa\\i + w'{Al3 - Ax) -^w'Aji/^j - xj) (18) 

= \\xa\\i -'^w' A j{f3j - fij + fij - Xj) (19) 

= I^aIIi - ^ w'Aj{l3j - Aj - - X] ~ + 

> \\xa\\i = \\xtA\i (21) 
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In the above, the inequaUty in ([T6l ) follows because w'Aj < \w'Aj\ < 1 for j ^ T U A and because 
> Inequality ( fTTl ) uses the fact that \z\ > sgn{b)z for any two scalars z and b and that Xj = for 
J ^ TU A. In (fTSl) . the first equality uses sgn(a;j)xj = |xj| and w'Aj = sgn(3;j) for j G A. The second 
equality just rewrites the second term in a different form. In ( fT9l ). we use the fact that Ap = Ax = y 
(since both /3 and x are feasible) to eliminate w'{A/3 — Ax). Equation ( [20l ) uses w'Aj = for j G Tin 
and the definitions of Ta+ and Ta_ given in ([T2l ). Finally, (|2T]) follows because — X]jeTi.+ ~ ~ 

p) — J2jeT^ '^'^jif^j ~ + p) > 0- This holds since —p < (3j — flj < p for aU j G T; if'^j > for 
j G Ta^; and w'Aj < for j G T^.. 

Both inequalities ( fTSl ) and ([T6l)-(l2TI) can hold only when ||/3t<=||i = lkT<=||i> i-C- all the inequalities in 
(O-lllT]) hold with equaUty. Consider the inequaUty in (O. Since \w'Aj\ < 1 for j ^ TU A, this holds 
with equahty only if /3j = for all j ^ TUA. Since A^ = y = Ax and since both /3 and x are supported 
on TU A (or on its subset), Atua{(3tuA — xtua) = 0. Since 6k+u < 1> ^TuA has full rank. Therefore, 
this means that PtuA = a^ruA- Thus, we can conclude that /5 = x, i.e., x is the unique minimizer. 

B. Proof of Lemma |2] 

This proof uses the following simple facts. Let Amin(-^'^)> ^ra&x{M) denote the minimum and maximum 
eigenvalues of a matrix M. (i) For positive semi-definite matrices, M, Q, \\M\\ = Xmax{M); \\MQ\\ < 
||Af||||Q||; Amin(Af — Q) > Amin(Af) — Amax(Q); and for a positive definite matrix, M, ||M~-^|| = 
l/Amin(M); (ii) for any matrices, S,C, ||-B-C|| < ||S|| + ||C||; (iii) for disjoint sets Ti, Pt/^tJ| < 
0\T^\,\n\ 11 equation (3)]; (iv) 1 - S\t,\ < A^in (^t/^tJ < X^m^MrMr,) < 1 + 5|Ti| II; (v) M{n) 
is a projection matrix and so M{Tb)M{Tby = M(Tb) and ||M(rf,)|| = 1; (vi) ||sgn(xA)||2 = Vu. 

The lemma assumes that 5„ + 6k^ + m ^ ^- This implies that (a) 6u < I and so A/\'A/\ is positive 
definite and so u < n; (b) 6^^ < 1 and so AtJAt^ is positive definite and M{T},) is well-defined; and 
(c) as we show next, A^ M{Tb)A^ is positive definite and hence full rank. Since A/^' M{Ti)A/^ = 
j4a'j4a — A^ AT^{ATf,' At^)~^ AtJ A/^ is a difference of two positive semi-definite matrices, thus, 

q2 

\unn{AA'M{n)A^) > XminiAA'A^) " Xm^^A AtMu AtJ'^ Aa) > (1 " <5J - > 

Thus, Aa' M{Ti,)Aa is positive definite. The first inequality in (l22l ) follows from fact (i). The second 
one follows because Amin(^A'^A) > (1 - 4) (using fact (iv)); Amax(^A'^T, (^t/^tJ"^^t/^a) = 
\\AA'ATMn'An)-^M'AA\\ < \\Aa'AtA\ WiArMrJ-'W UtMaW (using fact (i)); Pa'^tJI = 
Pt/^a|| < Ok,,u (using fact (iii)); and ||(^t/^tJ-^|| = a„„„(aI,'A^j ^ (^in^e AtMt, is 

positive definite, this follows using fact (i) and fact (iv)). The third inequality of (l22l ) follows because 

(1 — 6u) — iz^f" = — ''i-sl '''' — ^ ^ 0- Both the numerator and the denominator are positive 

because we have assumed that 6u + Sk^ + O^^ ^ < 1. 

Using fact (v), ^A'M(rfc)^A = AA'M{Tb)M{TbyAA. Thus, using the above, AA'M(Tb)M{TbyAA 
is positive definite and hence has full rank u. Thus, the u x n fat matrix, AAM{Tb) has full rank, u. 

To prove the lemma, we first try to construct an ?i x 1 vector, w, that satisfies the first two conditions 
of the lemma. Then, we show that we can find an exceptional set E so that the constructed w and 
E satisfy all the required conditions. Any w that satisfies AtJw = lies in the null space of AtJ 
and hence is of the form w = M{Tb)^. To satisfy the second condition, we need a 7 that satisfies 
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AA'M{Tb)^ = sgn(xA)- As shown above, A/\'M{Tb) is full rank and so this system of equations 
has a solution (in fact has infinitely many solutions). We can compute the minimum £2 norm solution 
in closed form as 7 = MiTb)' A a{A a' M{Tb)M{Tby A Ay^sgn{x a). Since M{Ti,)M{Tby = M{Tb), 
w = M{Ti,)'y can be rewritten as 



w 



M in) AAiAA'MiTb) A A)-hgn{x a) (23) 



Using the definition of T.^+g, T^-g given in (fT4l ) in Theorem 1, we can see that w satisfies the first two 
conditions of the lemma. Recall that Ai'w > for all i G Ta+g is equivalent to At^^^'w >- 0, and similarly, 
Ai'w < for all i G T^.g is equivalent to At._Jw -< 0. 

The rest of the proof is similar to that of (T, Lemma 2]. Consider any set disjoint with T U A of 
size |Trf| < s. Then, 

WAfJwh < \\AfjM{n)AA\\ \\{AA'M{n)AA)-^\\ ||sgn(xA)||2 

< (^s,« + %4^) ^-^^V^ = ak,iu,s)V^ (24) 

Notice that ak^{u,s) is positive because we have assumed that + 6^^ + ^fc, « < 1- The bound in 
(l24l) follows using the simple facts given in the beginning. We obtain (l24l) as follows. Consider the first 
term \\AfjM{Tb)AA\\. Using the definition of M{Tb) and fact (ii), \\AfjM{Tb)AA\\ < \\AfjAA\\ + 
WAfjATMrjATj-^ATjAAl Using fact (iii), Pf/^A|| < Os,u, Pf/^nll < ^.-.fc, and Pt/^a|| < 
9u,k,- Since At,' At, is positive definite, using fact (i) and fact (iv), \\{At,' At,)'^\\ = A.„.„(AT,'ArJ - 
-^T-. Thus, we get \\Aj. 'M{Tb)AA\\ < {Os u+^^t=t^)- Consider the second term \\{Aa' M{Tb)AA)~'^\\. 

J- Ofcf, -fd 'J- Ok, 

Since AA'M{Tb)AA is positive definite, using fact (i) and (ED, \\{Aa'M {Tb)AA)~-^\\ = a {A^'M(t,)A^) - 
3—^2 — ■ Using fact (vi), the third term, ||sgn(xA)||2 = \/u. 



Define the set, E, as E := {j G (T U A)^ : \Aj'w\ > ^iti^jh/^}. Notice that \E\ must obey 
\E\ < s since otherwise we can contradict (l24l ) by taking 7^ C Since \E\ < s and E is disjoint 
with TU A, dill) holds for T^; = E, i.e., ||A£;'iD||2 < afej,(n, s)^. Also, by definition of E, \Aj'w\ < 
ak,iu^y/u ^ all j ^ T U Au E. Thus w satisfies the third condition of the lemma. 

Finally, \\w\\2 < ||M(rfe)|| Pa|| \\{Aa' M{Tb)AAy'^\\ < Kk,{u)y/^. This follows using fact 
(v); II^aII < \/ 1 + f^u ; and fact (i) and (l22l ). Thus, we have found a w and that satisfy all required 
conditions. 
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Supplementary Material 

C. Proof of Theorem |7] 

We construct a w that satisfies the conditions of Lemma [T] by first applying Lemma |2] and then applying 
Lemma [3] iteratively as explained below. Finally we define w using (|29l ) below. At iteration zero, we 
apply Lemma |2] with s = u. Lemma |2] can be applied because kb < k and 6u + Sk + 0^ ^ < I (holds 
because condition [T] of the theorem holds). Thus, there exists a wi and an exceptional set T^ i, disjoint 
with T U A, of size less than s = u, s.t. 

Aj'wi > 0, y j £ Ta+g 

A/wi < 0, V i G ra_g 
A/wi = o,y jen 

A/wi = sgn(xj), V j G A 
\Td,i\ < u 

\Aj'wi\ < ak,{u,u), Vi^TuAuTd,! 
Ikilb < KkSu)./ll (25) 

At iteration r > 0, apply Lemma[3]with = AuT^^r (so that s = 2n), = V j G A, Cj = A/wr V j G 
Trf^ and s = u. Call the exceptional set Td^+i- Lemma [3] can be applied because 62U + + ^1 2u < ^ 
(condition [U of the theorem). From Lemma [3l there exists a Wr+i and an exceptional set T^ y+i, disjoint 
with T U A U Td^r, of size less than s = u, s.t. 

^j'tfr-l-l = V j G T 
^/-Wr+i = 0, V i G A 

^/-Wr+l = Aj'wr, V J G Trf^r- 
\\AT^„^^'Wr+l\\2 < ak{2u,u)\\AT^„'wr\\2 

, ak{2u,u) , 

\Aj Wr+l\ < ^ AT^^It'r 2 



Vj ^ T U A U Td,r U rrf,,+i 

||lf^r+l||2 < Kki2u)\\AT^Jwr\\2 (26) 

Notice that |T^,i| < u (at iteration zero) and IT^; < u (at iteration r) ensures that lAuT^^^I < s = 2u 
for all r > 1. 
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The last three equations of (I26b . combined with the sixth equation of (1251) . simplify to 

II^Td,.+i''w^r+i||2 < ak{2u,uy'ak,iu,u)y/u 
\A/wr+i\ < ak{2u,uYakSu,u), 

yj^TuAU Td,- U Td,r+i (27) 
Ikr+ilb < Kk{2u)ak{2u,uY~^ak,{u,u)^/u 



(28) 



We can define 



oo 

A 



W 



Y^^-iy-^Wr (29) 



Since ak{2u, u) < 1, \\wr\\2 approaches zero with r, and so the above summation is absolutely convergent, 
i.e. w is well-defined. 

From the first four equations of (1251 ) and first two equations of (l26l ). 

A/w > 0, V j G Ta+g 

A/w < 0, V j G Ta-g 

A/w = 0, V J G Tb 

A/w = A/wi = sgn(xj), V J G A (30) 

Consider Aj'w = A/ Y^=\{~^^~^'^t for some j ^ T U A. If for a given r, j G Td,r> then A/wr = 
A/wr+i (gets canceled by the r + 1*^ term). If j G T^^r-i^ then A/wr = A/wr-i (gets canceled by 
the r — I*'' term). Since and T^^r-i are disjoint, j cannot belong to both of them. Thus, 

A/w= i-iy-^A/wr, ^TU A (31) 

Consider a given r in the above summation. Since j ^ T^ r U T^^r-i U T U A, we can use (l27l) to get 

l^/iL'rl < «fc(2ii, ii)''~^afc^(n, n). Thus, for all j ^ T U A, 

< ^ ak{2u,uY~^ak,iu,u) 

r:jiTa,rUTa,r-i 

~ l-afc(2n, n) 
Since ak{2u,u) +ak^{u,u) < 1 (condition |2] of the theorem), 

\A/w\ < 1, Vj ^ TU A (33) 

Thus, from ( [30l ) and ( [33] ). we have found a it; that satisfies the conditions of Lemma [T] From condition 
[T]of the theorem, (J^+m < 1. Applying Lemma [T] the claim follows. ■ 
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D. Proof of Lemma \3\ 
Let M = M{T). 

The lemma assumes that Ss + 5k + 0"^ ^ < 1. This means that (a) 5k < 1 and so At' At is positive 
definite; (b) 5s < 1 and so for any set of size \Tci\ < s, AtJAt^ is positive definite; and (c) 
as we show next, for any set of size |T^| < s, AtJMAt^ is also positive definite. Notice that 
AtJ MAt^ = AtJAtj^ — AtJ At{At' At)~^ At' At^ which is the difference of two symmetric non- 
negative definite matrices. Let Bi denote the first matrix and B2 the second one. Use the fact that 

Amin(5l - B2) > Amm(^l) + Amin(-S2) = Amin(-Bl) " Amax(-B2) whcrC Amin (•) > Amax (•) dcnOtC the 

\\(A 'A )\\^ 

minimum, maximum eigenvalue. Since Amin(-Bi) > (1 — Sg) and Amax(-B2) = II-B2II < — 
-, thus 



1-4 

XminiATjMAT,) > I - 5s - > (34) 

I - Ok 

(the last inequality holds because 5s + 5k + Of ^ < 1). Thus, AtJMAt^ is positive definite. 

Since M is a projection matrix, MM' = M, and so AtJMAt^ = AtJMM'At^. Thus, from above, 
AtJMM'At^ is also positive definite. Thus, AtJM is full rank. 

Any w that satisfies At'w = will be of the form 



w 



[I - AT{AT'AT)~^AT'h := M-f (35) 



We need to find a 7 s.t. AtJw = c, i.e. AtJM^ = c. Since AtJM is full rank, this system of 
equations has a solution (in fact, it has infinitely many solutions). Let 7 = M'At^V- Then rj = 
{AtJMM'At^)-^c = {AtJMAt^Y^c. This follows because MM' = M^ = M since M is a projection 
matrix. Thus, 

w = MM'AtMtJMAt^)-^c = MAtMtJMAtX^c (36) 
Consider any set with |Trf| < s disjoint with TU T^. Then 

Ufjwh < WA^JMAtA \\{AtJMAt,)-^\\ \\c\\2 (37) 
Consider the first term from the right hand side (RHS) of (l37l l. 

WA^JMAtA < \\AfjATA + \\AfjAT{AT'AT)-^AT'ATA 

< es,s + ^^^^ (38) 

1 - C>fc 



This follows in a fashion exactly analogous to the derivation of the upper bound on the first term of (1241 ) 
in the proof of Lemma 2. Consider the second term from the RHS of (|37l ). Since AtJMAt^ is positive 
definite, 

\\{AtJMAt,)-'\\ = \ (39) 

Amm(^T<i MAtJ 



\\{At:MAt,)-^\\ < — (40) 



Using dMJI, 
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Recall that the denominator is positive because we have assumed that + + s ^- Using (1381 ) and 
(|40l ) to bound ^T}, we get that for any set with \fd\ < s, 

a a 

1 



\Af^w\\2 < ' ||c||2 =flfc(3,s)||c||2 (41) 



' 1-5, 

Notice that 0^(5, s) is non-decreasing in k, s, s. Define an exceptional set, E, as 

E := {j G {TUTaY : \A/w\ > ^^%^||c||2} (42) 

VS 

Notice that \E\ must obey |£^| < s since otherwise we can contradict (|4T] ) by taking C 

Since \E\ < s and E is disjoint with TUTa, dH) holds for = E, i.e. ||As''«;||2 < afe(s, s)||c||2. 
Also, by definition of E, \Aj'w\ < ^^^||c||2, for all j (j^TUTdU E. Finally, 

V ^ 

\\w\\2 < \\MAtMtJMAt,)-^\\ \\c\\2 

< \\M\\ \\AtA\ \\{At/MAt,)-A\ \\ch 

< —^^^^^^\\c\\2 = Kkis)\\c\\2 (43) 
^ l-5k 

since ||M|| = 1 (holds because M is a projection matrix). Thus we have found a w and a set E that 
satisfy all conditions of the lemma. 
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